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(54) A method of annotating an image 

(57) A method and apparatus for annotating an im- 
age (407) is disclosed. The image (407) and a plurality 
of icons (403) are displayed such that each icon is as- 
sociated with metadata. At least one of the icons is se- 
lected depending on at least one subject of the image 
(407) and the metadata associated with the selected 
icon is stored as an annotation of the subject of the im- 
age. 
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Description 

Technical Field of the invention 

[0001] The present invention relates generally to the 
field of multi-media and in particular, to a method and 
apparatus for generating metadata based on multi-me- 
dia content. The invention also relates to a computer 
program product including a computer readable medi- 
um having recorded thereon a computer program for 
generating metadata based on multi-media content. 

Background Art 

[0002] The popularity of digital photography Is ever in- 
creasing as digital cameras become more accessible 
and the production and editing capabilities of digital im- 
ages improve. Further, the Internet use of digital Images 
is increasing rapidly. Large databases of digital images 
are being assembled for both personal and commercial 
use. As with conventional photography, the need to an- 
notate and catalogue the ever Increasing number of dig- 
ital images is of paramount importance in order to allow 
ease of access and use. One method of facilitating the 
annotation of digital images Is to generate "metadata" 
with the image. Metadata is infonnation about the con- 
tent of digital images or even video. For example, an 
image depicting a beach scene could include a short 
textual description such as "a picture of a beach", the 
name of a person in the image or a date and time that 
the Image was captured. Many Internet image search 
sites search on metadata content descriptions to locate 
digital images for display. 

[0003] Some digital cameras automatically generate 
a date and time, which is generally Included in the file 
name of a digital Image when the image is stored and/ 
or displayed (e.g. 12Nov_1 .jpg). However, the automat- 
ically generated date and time says nothing about the 
content and/or event depicted by the digital image and 
therefore provides only limited assistance In annotating, 
cataloguing and searching for the digital Image. 
[0004] Conventionally, a text entry method of gener- 
ating metadata for digital images has been used where 
a person sorts through a database of digital images, us- 
ing a computer, and stores with each digital Image a 
short textual label indicating a subject and/or an event 
depicted by each digital image. However, the above 
conventional method is very labour intensive and thus 
time consuming. 

[0005] The problem is further exacerbated when a 
person needs to supply a selection of digital Images of 
a particular event and/or subject by e-mail (electronic 
mail). This Involves viewing all of the images in a data- 
' base, selecting the appropriate images and then group- 
ing the images together by attaching them one-by-one 
to an e-mail for sending. 



Disclosure of the Invention 

[0006] It is an object of the present invention to sub- 
stantially overcome, or at least ameliorate, one or more 
5 disadvantages of existing arrangements. 

[0007] According to one aspect of the present inven- 
tion there Is provided a method a method of annotating 
an image, said method comprising the steps of:, 

10 displaying the image and a plurality of icons, each 

Icon being associated with metadata; 

selecting at least one of said icons depending on at 

least one subject of the Image; and 

storing the metadata associated with said selected 
15 icon as an annotation of the subject of the image. 

[0008] According to another aspect of the present in- 
vention there is provided an apparatus for annotating an 
image, said apparatus comprising: 

20 

display means for displaying the image and a plu- 
rality of icons, each Icon being associated with 
metadata; 

selection means for selecting at least one of said 
25 icons depending on at least one subject of the Im- 
age; and 

storage means for storing the metadata associated 
with said selected icon as an annotation of the sub- 
ject of the image. 

30 

Brief Description of the Drawings 

[0009] A number of embodiments of the present in- 
vention will now be described with reference to the draw- 
35 ings, in which: 

Fig. 1 is a flow diagram showing a method of anno- 
tating an image using metadata, in accordance with 
a preferred embodiment of the present invention; 

40 Fig. 2 is a schematic block diagram of a general pur- 
pose computer upon which anrangements de- 
scribed can be practiced; and 
Fig. 3 shows an example fonnat of an XML file;. 
Fig. 4 shows a user Interface that can be used to 

45 perfomi the method of Fig. 1 ; 

Fig. 5 shows the user interface of Fig. 4 with bound- 
ing boxes overiaying the image according to the 
method of Fig. 1 ; and 

Fig. 6 is a flow diagram showing a method of anno- 
50 tating an image using metadata according to a fur- 
ther embodiment of the present Invention. 

Detaiied Description including Best IIAode 

55 [0010] Where reference is made in any one or more 
of the accompanying drawings to steps and/or features, 
which have the same reference numerals, those steps 
and/or features have for the purposes of this description 
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the same function(s) oroperation(s), unless the contrary 
intention appears. 

[001 1 ] The principles of the method described herein 
have general applicability to generating metadata. How- 
ever, for ease of explanation, the steps of the method s 
are described with reference to digital images. However, 
it is not intended that the present invention be limited to 
the described method. For example, the invention may 
have application to generating metadata regarding the 
subject of a document, a video clip or any computer gen- io 
erated page. 

[0012] Fig. 1 is a flowchart 100 showing a method of 
annotating an image using metadata, in accordance 
with the preferred embodiment of the present invention. 
The method of Fig. 1 is preferably practiced using a con- 15 
ventional general-purpose computer system 200, such 
as that shown in Fig. 2 wherein the processes of Fig. 1 
may be implemented as software, such as an applica- 
tion program executing within the computer system 200. 
In particular, the steps of the method of annotating an 20 
image using metadata, are effected by instructions in 
the software that are carried out by the computer. The 
software may be divided into two separate parts; one 
part for carrying out the annotation method; and another 
part to manage the user interface between the method 
and the user The software may be stored in a computer 
readable medium, including the storage devices de- 
scribed below, for example. The software is loaded into 
the computer from the computer readable medium, and 
then executed by the computer. A computer readable so 
medium having such software or computer program re- 
corded on it is a computer program product. The use of 
the computer program product in the computer prefera- 
bly effects an advantageous apparatus for annotating 
an image using metadata in accordance with the em- 35 
bodiments of the invention. 

[0013] The computer system 200 comprises a com- 
puter module 201 , input devices such as a l<eyboard 202 
and mouse 203, output devices including a printer 215 
and a display device 214. A Modulator-Demodulator 40 
(Modem) transceiver device 21 6 is used by the compu- 
ter module 201 for communicating to arid fronfi a corri- 
munications networt< 220, for example connectable via 
a telephone line 221 or other functional medium. The 
modem 21 6 can be used to obtain access to the Intemet, ^ 
and other network systenns, such as a Local Area Net- 
wort( (LAN) or a Wide Area Network (WAN). 
[0014] The computer module 201 typically includes at 
least one processor unit 205, a memory unit 206, for ex- 
ample fomned from semiconductor random access so 
memory (RAM) and read only memory (ROM), input/ 
output (I/O) interfaces including a video Interface 207, 
and an I/O interface 213 for the keyboard 202 and 
mouse 203 and optionally a joystick (not illustrated), and 
an interface 208 for the modem 21 6. A storage device 55 

209 is provided and typically includes a hard disk drive 

210 and a floppy disk drive 211 . A magnetic tape drive 
(not illustrated) may also be used. A CD-ROM drive 212 



is typically provided as a non-volatile source of data. The 
components 205 to 213 of the computer module 201. 
typically communicate via an interconnected bus 204 
and in a manner which results in a conventional mode 
of operation of the computer system 200 known to those 
in the relevant art. Examples of computers on which the 
embodiments can be practised Include IBM-PC's and 
compatibles, Sun Sparcstations or alike computer sys- 
tems evolved therefrom. 

[0015] Typically, the application program of the pre- 
ferred embodiment is resident on the hard disk drive 21 0 
and read land controlled in its execution by the processor 
205. Intemnediate storage of the program and any data 
fetched from the networi< 220 may be accomplished us- 
ing the semiconductor memory 206. possibly in concert 
with the hard disk drive 210. In some instances, the ap- 
plication program may be supplied to the user encoded 
on a CD-ROM or floppy disk and read via the corre- 
sponding drive 212 or 211 , or alternatively may be read 
by the user from the networi< 220 via the modem device 
216. Still further, the software can also be loaded into 
the computer system 200 from other computer readable 
medium including magnetic tape, a ROM or integrated 
circuit, a magneto-optical disk, a radio or infra-red trans- 
mission channel between the computer module 201 and 
another device, a computer readable card such as a PC- 
MCIA card, and the Intemet and Intranets including e- 
mail transmissions and infonnatlon recorded on web- 
sites and the like. An e-mail transmission is an electronic 
message sent from one computer user to another over 
a digital network such as the Intemet or Intranet. The 
message is generally sent using an e-mail application 
program and is generally In the fomn of a text file con- 
taining a textual message. However, an e-mail transmis- 
sion can include other file types such as .jpg andUif 
files, for example. The foregoing is merely exemplary of 
relevant computer readable mediums. Other computer 
readable mediums may be practiced without departing 
from the scope and spirit of the invention. 
[0016] The method of annotating an image using 
metadata can alternatively be implemented in dedicated 
hardware such as one or more Integrated circuits per- 
fomiing the functions or sub functions of Fig. 1. Such 
dedicated hardware may include graphic processors, 
digital signal processors, or one or more microproces- 
sors and associated memories. 
[0017] The method of annotating an image using 
metadata, can now be described with reference to the 
flowchart 100 of Fig. 1 , where the method is perfomned 
using the computer system 200. The process begins at 
step 1 01 , where a list of metadata labels is provided. 
The list of metadata labels is preferably provided auto- 
matically. For example, a list of people's names can be 
provided automatically by extracting the names frorti an 
existing database of names, such as an e-mail address 
book. In this instance, names, associated e-mail ad- 
dresses and other personal information can be extract- 
ed from the database. Altematively, a list of metadata 
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labels can be manually generated using any known soft- 
ware database application, or metadata labels can be 
manually Inputted by the user using the keyboard 202. 
At the next step 103, a plurality of icons are provided. 
Preferably, the number of icons is equal to the number 
of metadata labels in the list of metadata labels and the 
Icons are preferably displayed on the video display 214. 
The process continues at the next step 1 05, where each 
label in the list of metadata labels Is associated with at 
least one of the generated Icons. For example, one of 
the icons could be associated with the name "Jenny 
Smith" which was included In an Imported e-mail ad- 
dress book. The association of each metadata label with 
an icon can be earned out arbitrarily. However, prefera- 
bly each metadata label is associated with a generated 
icon. 

[0018] At the next step 107, a digital image is ren- 
dered. The digital image Is preferably rendered on a dis- 
play, such as the video display 214. The digital image is 
preferably extracted from a database of images which 
is stored on the hard disk drive 21 4 or accessed via the 
computer network 220. The process continues at the 
next step 109, where one of the generated Icons is se- 
lected by the user according to the content of the image. 
For example, if the image depicts a person called "Liza 
Hayward", then the icon associated with the name Liza 
Hayward can be selected. The Icon is preferably select- 
ed by clicking on the appropriate icon using the mouse 
203. The icon can preferably be dragged in a conven- 
tional manner using the mouse 203 and positioned over 
a desired subject of the image. Once the icon is posi- 
tioned over the desired subject of the image, the user 
can de-select the mouse 203 in the conventional man- 
ner. This action is generally referred to as "dropping" the 
Icon on the image. At the next step 111 , the location at 
which the subject is rendered in the image is determined 
based on the initial placement of an icon on an image. 
Any known face or object detection method (e.g. colour 
region growing) can be used to detect where the subject 
Is located In the Image based on the location of the Initial 
placement of the icon. In accordance with the preferred 
embodiment, a default sized bounding box (e.g. 503 of 
Fig. 5) is presented to the user based on the location of 
the initial placement of the icon such that the bounding 
box substantially sun^ounds the subject (e.g. Liza Hay- 
ward). The bounding box Is preferably re-sizeable by the 
user using the mouse 203 in a conventional manner. In 
this manner, the user can alter the size of the bounding 
box to totally surround the selected subject or to cover 
only a portion of the subject. Preferably, more than one 
loon can be selected. Alternatively, the bounding box 
can be automatically resized to fit the selected subject. 
For example, if there Is more than one person in the ren- 
dered image then another icon, which is associated with 
the name of the other person, can be selected, dragged 
and positioned over the other person, as discussed 
above. This will result in another default sized bounding 
box (e.g. 505 of Fig. 5) being positioned over the second 



subject, which can preferably be re-sized by the user, 
as discussed above. In another example, a person in 
the image can be standing with a mountain in the back- 
ground. Therefore, after selecting the icon associated 
s with the person's name, an icon associated with the 
metadata "mountain" can be selected and if desired a 
bounding box can be positioned adjacent to the moun- 
tain. 

[0019] At the next step 113, the metadata (e.g. the 

10 name "Liza Hayward") associated with the selected 
icons is stored as an association list, in a storage device 
such as the hard disk drive 21 0, and linked to the ren- 
dered image. The position (x,y) and size (width, height) 
of the bounding box (e.g. 503), associated with the sub- 

15 ject, are also stored in the association list, as at step 
113, such that the metadata and bounding box Informa- 
tion are linked together. Alternatively, two opposite co- 
ordinate points indicating the size of the bounding box 
can be stored in the association list. Therefore, the 

20 bounding box information (i.e. position, height and 
width) indicates the location and size of the subject with- 
in the rendered digital image. The association list pref- 
erably includes a tag indicating an association with the 
rendered image. For example, the list could include the 

25 file name (e.g. 12Nov_1 .jpg) of the corresponding im- 
age file for the rendered image. Preferably the associa- 
tion list is an Extensible Mark-up Language (XML) file, 
stored on a hard disk drive, such as the hard disk drive 
210. XML is a computer language used for associating 

30 metadata with images. An example of the fomriat of an 
XML file 301 is shown in Fig. 3. As seen in Fig. 3, the 
XML file includes the file name 300 of an image file, the 
metadata 303, and bounding box infonnation 305 asso- 
ciated with the image file. In a further implementation., 

35 the metadata and bounding box positional Infonnation' 
associated with the selected subject can be stored as 
part of the image file. For example, the .TIF image for- 
mat allows metadata to be stored as part of the Image 
file. 

40 [0020] The process continues at step 115, where a 
decision is made as to whether any more images require 
annotation. If more Images require annotation, then the 
process returns to step 1 07, where another image is ren- 
dered. The process concludes when all of the images 

45 requiring annotation have been annotated. 

[0021 ] Fig. 4 shows a preferred user interface 401 that 
can be used to perfomn the method shown in Fig. 1 . The 
preferred user interface 401 includes an icon 415 la- 
belled "Input Icon" which can be selected in order to en- 

so able the user to generate a database of metadata labels, 
as at step 1 01 of the method, by importing a pre-existing 
database from the computer network 220 or another ap- 
plication. The preferred user interface 401 also Includes 
an icon 417 labelled "Import Images" which can be se- 

55 lected in order to enable the user to access the digital 
Images to be rendered. The user can preferably import 
the digital images from a pre-existing database which is 
stored on the hard disk drive 214 or accessed via the 
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computer network 220 or another application. The user 
interface 401 also Includes a series of icons 403, which 
are generated as at step 1 03 of the method. One of the 
icons 403 can be selected, as at step 1 09 of the method, 
in order to associate the selected icon with a rendered s 
image. The number of generated icons 403 is preferably 
equal to the number of metadata labels In the generated 
database, which has been previously imported or cre- 
ated by the user. The user Interface 401 also includes 
a frame 407 for rendering the images that the user wish- io 
es to annotate. A forward and reverse button 409 and 
411 , respectively, are provided below the frame 407 to 
enable the user to successively render alt of the images 
Included on the database by selecting either forward or 
reverse with the mouse 203 in a conventional manner, 
[0022] In accordance with a further embodiment, the 
user interface 401 includes three icons 419, 421 and 
423 labelled "Extract Sub-Images", "Display Sub-imag- 
es" and "Rectangles", respectively. The user can select 
the icon 419 labelled "Extract Sub-images" resulting in 20 
an image being extracted from the rendered image, 
based on a bounding box which has been dropped on 
the rendered image using the method of Fig. 1 . Also, 
upon selecting the icon 41 9 labelled "Extract Sub-imag- 
es", the bounding box infomriation (i.e. position, height 
and width) associated with the bounding box is stored 
in the association list together with the metadata asso- 
ciated with the subject overiayed by the bounding box. 
After being extracted, the Image in the bounding box can 
preferably be stored as an image file (e.g .jpg or.tif) in 
the semiconductor memory 206. For example, an Im- 
age, depicting Liza l-layward, based on the bounding 
box positioned over Liza Hayward In the rendered Im- 
age, Is extracted and stored in the semiconductor mem- 
ory 206. and the position and size of the bounding box 
are stored in the association list together with the meta- 
data associated with Liza Hayward, upon the user se- 
lecting the icon 419 labelled "Extract Sub-images". As 
can be seen in Fig. 5, three bounding boxes 501, 503 
and 505 have been displayed on the image being ren- 
dered in the frame 407 of the user interface 401 when 
the user selects the icon 421 labelled "Display Sub-Im- 
ages". The user can then select the icon 421 labelled 
"Display Sub-images" which results in the extracted im- 
age being displayed on the user interface 401 . A user 
has then selected the Icon 419 labelled "Extract Sub- 
images" followed by the icon 421 labelled "Display Sub- 
images" which has resulted in three Images 507, 509 
and 511 being displayed on the user interface 401 based 
on the bounding boxes 501 , 503 and 505. The process 
of extracting the sub-images In accordance with the fur- 
ther embodiment at steps 109, 111 and 113 can be car- 
ried out sequentially or concurrently after dropping the 
selected icon 403. 

[0023] Selecting the icon 423 labelled "Rectangles" 
preferably results in the bounding boxes (e.g. 501 , 503 
and 505) being removed from the rendered image. How- 
ever, the selection of the icon 423 does not result in the 



removal of previously stored bounding box infomriation 
and metadata associated with the image. Re-selecting 
the icon 423 preferably results in the display of all 
bounding boxes that have previously been associated 
with an image. For example, if a user selects the forward 
button 409 and the image displayed in Fig. 4 is selected 
from a database of images and rendered in the frame 
407, then a user can select the icon 423 resulting In all 
previously stored bounding boxes (e.g. 501 , 503, and 
505) being displayed. Re-selecting the icon 423 prefer- 
ably results In the bounding boxes being removed from 
the image. However, the associated bounding box infor- 
mation and metadata previously stored (n the associa- 
tion list associated with the image will not be affected. 
[0024] In accordance with a still further embodiment 
of the present invention, once bounding box positional 
infomriation and metadata have been stored for an im- 
age, pointing to a subject In the Image with a mouse 
pointer (not illustrated) in a conventional manner will re- 
sult in the metadata associated with the subject being 
displayed. As seen in Fig. 5, a user has selected Liza 
Hayward with the mouse pointer after the bounding box 
503 has been dropped onto the image, resulting in the 
display of the "tip" 515 including Liza Hayward's name. 
[0025] In accordance with a still further embodiment 
of the present invention, the user interface 401 can In- 
clude a "content" button (not illustrated) and a "like to 
send" button (not illustrated). The content button can be 
selected by a user before selecting one of the icons 403 
in order to indicate that the icon to be subsequently sot 
lected will be refen-ing to the content of the image. Air 
ternatively, the user can select the like to send button in 
order to indicate that the content of the image will be of 
interest to a selected person. The icon subsequently se- 
lected, after selecting the like to send button, will be as- 
sociated with the name of the person to whom the user 
wishes to send the image. Selecting the content button 
preferably results in a tag (not Illustrated) being included 
in the association list stored at step 113, Indicating that 
the metadata associated with the subsequently selected 
icon refers to the content of the image. Similariy, select- 
ing the like to send button preferably results in a tag (not 
illustrated) being included in the association list stored 
at step 1 1 3, indicating that the metadata associated with 
the subsequently selected icon refers to a person to 
whom the user wants to send the Image to. 
[0026] The icons that are provided at step 1 03 of the 
preferred method, are preferably default icons having a 
generic indicator displayed in them. For example, stick 
figures depicting generic people can be displayed in the 
default icons, as seen in Figs. 4 and 5. However, in ac- 
cordance with the preferred embodiment, if the image 
rendered at step 107 is subsequently only associated 
with one subject at step 1 09, then the icon used to indi- 
cate the association can have the generic indicator re- 
placed by a scaled and/or cropped version of the ren- 
dered image. For example, if the rendered image de- 
picts a person Jenny and the Icon associated with the 
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metadata "Jenny" is subsequently selected, then the ge- 
neric Indicator of the selected Icon can be replaced with 
a scaled. and/or cropped version of the image depicting 
Jenny. Alternatively, the extracted image resulting from 
a bounding box being dropped onto Jenny In an image 
can be depicted In the icon associated with the metadata 
"Jenny". Replacing the default Icons with images Im- 
proves the user friendliness of the icons 403. 
[0027] In accordance with stilt a further embodiment 
of the present invention, the icons provided at step 1 03 
can have e-mail addresses associated with them at step 
105. In this case, the database of metadata labels, cre- 
ated or imported by the user at step 1 01 , Includes an e- 
mait address associated with each name Included in the 
database. When an icon is subsequently selected, at 
step 109 of the method, the metadata associated with 
the selected icon, describing the person's name and e- 
mail address, is stored in the association list at step 113 
of the method. Subsequently, if a user wishes to e-mail 
a number of Image files to another person, the e-mail 
application can search through the association lists as- 
sociated with each image file annotated by the user and 
then send the appropriate images to the person. For ex- 
ample, the BHTiail application can search through the as- 
sociation lists and send all of the images that have the 
metadata "Jenny" associated with them using Jenn/s 
e-mail address, in a further example, if several of the 
lists have been tagged as "like to send", as described 
above, then the e-mall application can select those im- 
ages and e-mail them to the address of the person indi- 
cated by the metadata Included in the association lists 
associated with those images. 
[0028] Once an image or document has been anno- 
tated using the method, a user can perform any number 
of functions with the annotated material. For example, 
a user can collate a number of annotated images for 
printing or faxing. Further, a number of annotated Imag- 
es can be displayed in sequence based on the content 
of the images. Still further, the annotation can facilitate 
In later searching for and recovering the image. Still fur- 
ther, the annotation can facilitate In the automatic pres- 
entation of images allowing a user to zoom in on a sub- 
ject of interest. Still further, the annotation can facilitate 
in the generation of "tool tips" such that when a user 
does a conventional mouse roll-over of an annotated Im- 
age, an unknown person in the Image can be identified 
by pointing the mouse pointer at that person. Still further, 
the extraction and display of Images In accordance with 
the pretended implementation can be used to subdivide 
and/or mosaic an image. 

[0029] Fig. 6 Is a flow diagram 600 showing a method 
of annotating an image using metadata, in accordance 
with a still further embodiment of the present invention. 
The process of flow chart 600 begins at step 601 , where 
a list of metadata labels is provided. The list of metadata 
labels is preferably provided automatically. For exam- 
ple, as discussed above, a list of people's names can 
be automatically provided by extracting the names from 



an existing database of names, such as an e-mail ad- 
dress book. In this instance, names, associated e-mail 
addresses and other personal information can be ex- 
tracted from the database. Alternatively, a list of meta- 

5 data labels can be manually generated using any known 
software database application, or metadata labels can 
be manually inputted by the user using the keyboard 
202. At the next step 603, a plurality of icons are provid- 
ed. Preferably, the number of icons is equal to the 

10 number of metadata labels in the list of metadata labels 
and the icons are preferably displayed on the video dis- 
play 214. The process continues at the next step 605, 
where each label in the list of metadata labels is asso- 
ciated with at least one of the generated icons, similarly 

15 to step 1 05 of Fig. 1 , as discussed above. 

[0030] At the next step 607, a digital image is ren- 
dered on the display 214. The process continues at the 
next step 609, where a user can select that one or more 
subjects from the rendered Image be automatically de- 

20 tected. If automatic detection is selected at step 609, 
then the process of flow chart 600 proceeds to step 61 1 . 
Otherwise, the process proceeds directly to step 615. 
At step 611, subjects of the rendered image are auto- 
matically detected, using any known image detection 

25 method. For example, face detection or object detection 
can be perfomned using the colour Information of the im- 
age. In accordance with one embodiment, the distribu- 
tion of human face patterns within an image can be mod- 
elled by means of view-based "face" and "non-face" pro- 

30 totypes. At each location in the image, a difference fea- 
ture vector can be computed between the Image pattern 
and the distribution model for the image. A human face 
can then be detected at a particular location in the image 
based on the difference feature vector computed for that 

35 particular location. One such face detection method Is 
described In the article by Kah-Kay Sung and Tomaso 
Poggio, entitled "Example-based Leaming for View- 
based Human Face Detection", A.I. Memo No. 1521 , C, 
B.C.L. Paper No. 112, Artificial Intelligence Laboratory, 

40 Massachusetts Institute of Technology, 1994. Also at 
step 611 , rectangles (i.e. bounding boxes) surrounding 
the detected subjects, are automatically displayed on 
the image. At the next step 613, the rectangles dis- 
played at step 611 are adjusted to fit the detected sub- 

45 jects. The rectangles are preferably adjusted by being 
re-sized, moved or erased by the user using the key- 
board 202 and/or the mouse 203 In a conventional man- 
ner. 

[0031] The process of flow chart 600 continues at the 
50 next step 615, where if the user wishes to manually 
place a rectangle on the image with respect to any par- 
ticular selected subject of the image, the process pro- 
ceeds to step 61 7. Otherwise, the process proceeds di- 
rectly to step 623. At step 61 7, a default sized rectangle 
55 (e.g. 503 of Fig. 5) is placed over the selected subject. 
The process continues at the next step 619, where the 
default sized rectangle is adjusted by being re-sized, 
moved or erased by the user. At the next step 621 , if the 
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user wishes to manuaHy place any further rectangles on 
the image with respect to any particularly selected sub- 
ject of the image, the process returns to step 61 7. Oth- 
erwise, the process proceeds to step 623. 
[0032] At step 623, at least one of the plurality of Icons 
provided at the step 603. is selected by the user accord- 
ing to the content of the image, and the selected icon is 
dragged onto the image using the mouse 203 in a con- 
ventional manner At the next step 625, the rectangle 
surrounding the selected subject is automatically em- 
phasized when the dragged icon Is positioned over the 
selected subject. The rectangle is preferably empha- 
sized by changing its colour or brightness. At the next 
step 627, if the selected icon has been dropped onto the 
image with respect to the selected subject then the proc- 
ess proceeds to step 629. At step 629, the emphasized 
rectangle at the location where the icon is dropped, Is 
detected. The process continues at the next step 631 , 
where metadata associated with the dropped icon is 
stored as an association list in a similar manner to that 
discussed above with reference to step 11 3 of flow chart 
100. Preferably, two opposite coordinate points of the 
emphasized rectangle are stored with the metadata In 
the association list. At the next step 633, if the user wish- 
es to add another icon and associated metadata to the 
Image, then the process retums to step 623. Othenwise, 
the process proceeds to the next step 635, where if there 
are more Images requiring annotation, then the process 
retums to step 607, where another Image Is rendered. 
The process concludes when all of the images requiring 
annotation have been annotated. 
[0033] The aforementioned methods comprise partic- 
ular control flows. There are many other variants of the 
methods which use different control flows without de- 
parting the spirit or scope of the Invention. Furthemnore 
one or more of the steps of the methods can be per- 
formed in parallel rather than sequentially. 
[0034] The foregoing describes only some embodi- 
ments of the present invention, and modifications and/ 
or changes can be made thereto without departing from 
the scope and spirit of the invention, the embodiments 
being illustrative and not restrictive, in accordance with 
a still further embodiment of the present invention, 
based on the Initial placement of an icon on an image, 
any known face or object detection method can be uti- 
lised to detect where the subject associated with the 
Icon is in the image. Processing can then be earned out, 
using any l<nown segmentation method, to extract an 
exact spline outline of the subject resulting in positional 
Information being stored for the Image, as discussed 
above. 

[0035] In accordance with the still further embodi- 
ment, the bounding box, discussed above, can com- 
prise any number of edges. 

[0036] Further, the computer program for carrying out 
the method of the embodiments can be obtained in elec- 
tronic fonm for example by downloading the code over 
a networi( such as the intemet. Thus In accordance with 



another aspect of the present invention there Is provided 
an electrical signal carrying processor implementable 
instructions for controlling a processor to cany out the 
method as hereinbefore described. 

5 

Claims 

1. A method of annotating an Image, said method 
10 comprising the steps of: 

displaying the image and a plurality of icons, 
each icon being associated with metadata; 
selecting at least one of said icons depending 
15 on at least one subject of the image; and 

storing the metadata associated with said se- 
lected icon as an annotation of the subject of 
the image. 

20 2. A method according to claim 1 , wherein said select- 
ing step comprises the substeps of: 

selecting at least one of said icons depending 
on said at least one subject of the Image; 
25 dragging the selected Icon to the Image; 

dropping the dragged icon on the subject of the 
image; and 

detecting the subject based on the position at 
which the Icon is dropped, wherein said storing 
30 step stores the metadata associated with the 

dropped icon as an annotation of the subject of 

the Image. 

3. A method according to claim 2, wherein said detect- 
as Ing step extracts the subject from the image based 

on the dropped position when the dragged Icon Is 
dropped on the image. 

4. A method according to claim 3, wherein said subject 
40 is extracted based on the colour Infonrnation of the 

dropped position. 

5. A method according to claim 3, wherein said detect- 
ing step extracts a predetemnined sized region of 

45 the subject based on the dropped position and said 
storing step stores the metadata associated with 
the dropped icon as an annotation of the region of 
the subject. 

50 6. A method according to claim 2, further comprising 
the step of constructing regions about locations at 
which said subject Is rendered In said image. 

7. A method according to claim 6, wherein said detect- 
55 ing step detects one of the regions of the subject 
based on the position at which the icon is dropped, 
arid said storing step stores the metadata associat- 
ed with the dropped Icon as an annotation of the 
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detected region of the subject in the image. 

8. A method according to claim 6. wherein said drag- 
ging step further comprises the steps of: 

dragging the selected icon to the image; and 
emphasizing the region under the dragged 
icon. 

9. A method according to claim 1 , further comprising 
the step of constructing a bounding box about loca- 
tions at which said subject is rendered in said im- 
age. 

10. A method according to claim 9, further comprising 
the step of storing the location of said bounding box. 

11. A method according to claim 9, further comprising 
the step of extracting a part of said image based on 
the bounding box. 

12. A method according to claim 11 , further comprising 
the step of displaying the extracted part of said im- 
age. 

13. A method according to claim 9, wherein a size of 
said bounding box is detennined automaticaiiy. 

14. A method according to claim 9, wherein a size of 
said bounding box is changeable by a user. 

15. A method according to claim 1 , wherein said meta- 
data as the annotation of the subject is displayed 
upon selecting said subject in the image. 

16. A method according to claim 1, further comprising 
the steps of: 

providing a list of metadata; and 

associating the list of metadata and the plurality 

of icons. 

17. A method according to claim 16, wherein said list of 
metadata is provided from a database. 

1 8. A method according to claim 1 , wherein said storing 
step stores the metadata as the annotation of the 
subject of the image by using a tag indicating an 
association with said image. 

19. A method according to claim 18, wherein the meta- 
data associated with the subject of the image is 
stored in an XML file. 

20. A method according to claim 1 , further comprising 
the step of e-maiiing at least the Image to at least 
one e-mail address based on the metadata associ- 
ated with the image. 



21 . A method according to claim 1 , further comprising 
the step of replacing the default icon by the icon 
generated based on the subject of the image. 

5 22. A computer program for a computer, comprising 
software code portions for performing the steps of 
any one of claims 1 to 21 . 

23. A computer readable medium storing a computer 
10 program, wherein said computer program compris- 
es software code portions for perfomning the steps 
of any one of claims 1 to 21 . 

24. An apparatus for annotating an image, said appa- 
ls ratus comprising: 

display means for displaying the image and a 
plurality of icons, each icon being associated 
with metadata; 

20 selection means for selecting at least one of 

said icons depending on at least one subject of 
the image; and 

storage means for storing the metadata asso- 
ciated with said selected icon as an annotation 
25 of the subject of the image. 

25. The apparatus according to claim 24, further com- 
prising: 

30 means for dragging the selected icon to the im- 

age and dropping the dragged icon on the sub- 
ject of the Image; and 

detection means for detecting the subject 
based on the position at which the icon is 
35 dropped, wherein said storage means stores 

the metadata associated with the dropped icon 
as an annotation of the subject of the image. 

26. The apparatus according to claim 25. wherein said 
40 detection means extracts the subject from the im- 
age based on the dropped position when the 
dragged Icon Is dropped on the image. 

27. The apparatus according to claim 26, wherein said 
45 subject is extracted based on the colour information 

of the dropped position. 

28. The apparatus according to claim 25, wherein said 
detection means extracts a predetemnined sized re- 

50 gion of the subject based on the dropped position 
and said storage means stores the metadata asso- 
ciated with the dropped icon as an annotation of the 
region of the subject. 

55 29. The apparatus according to claim 25, further com- 
prising means for constructing regions about loca- 
tions at which said subject is rendered in said im- 
age. 
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30. The apparatus according to claim 29, wherein said 
detection means detects one of the regions of the 
subject based on the position at which the icon is 
dropped, and said storage means stores the meta- 
data associated with the dropped icon as an anno- 
tation of the detected region of the subject in the 
Image. 

31. The apparatus according to claim 29, wherein a re- 
gion under a dragged icon Is emphasized. 

32. The apparatus according to claim 25, further com- 
prising means for constructing a bounding box 
about locations at which said subject is rendered in 
said image. 



prising means for e-mailing at least the image to at 
least one e-mail address based on the metadata as- 
sociated with the image. 

5 44. The apparatus according to claim 24. further com- 
prising means for replacing the default Icon by the 
Icon generated based on the subject of the Image. 

45. A method as claimed in any of claims 1 to 21 where- 
10 in all of the steps recited therein are performed by 

a computer. 

46. A datastructure comprising image files, and respec- 
tive metadata, icon infomnation and bounding box 

15 information associated therewith. 



33. The apparatus according to claim 32, wherein said 
storage means further stores the location of said 
bounding box. 

34. The apparatus according to claim 32, further com- 
prising means for extracting a part of said image 
based on the bounding box. 

35. The apparatus according to claim 34, wherein said 
display means further displays the extracted part of 
said image. 

36. The apparatus according to claim 32, wherein a size 
of said bounding box is detennined automatically 
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47. An electrical signal comprising processor imple- 
mentable instmctions for carrying out a method as 
clairned in any one of claims 1 to 21 . 

48. A user interface for a computer system comprising 
icons associated with respective image files and 
with respective metadata, wherein selection of an 
icon selects the respective image file for processing 
in accordance with the respective metadata. 



37. The apparatus according to claim 32, wherein a size 
of said bounding box Is changeable by a user. 

38. The apparatus according to claim 24, wherein said ^ 
metadata as the annotation of the subject is dis- 
played upon selecting said subject in the image. 



39. The apparatus according to claim 24. further com- 
prising: 

means for providing a list of metadata; and 
association means for associating the list of 
metadata and the plurality of icons. 

40. The apparatus according to claim 39, wherein said 
list of metadata is provided from a database. 



40 



45 



41. The apparatus according to claim 24, wherein said . 
storage means stores the metadata as the annota- so 
tion of the subject of the Image by using a tag indi- 
cating an association with said image. 

42. The apparatus according to claim 41 , wherein the 
metadata associated with the subject of the image ss 
is stored in an XML file. 



43. The apparatus according to claim 24. further com- 
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<? xml version = "1.0" ?> 
<Entity 1 system 12Nov_l.jpg ND ATA jpeg> 



<Entity 2 system 1 2Nov_2.jpg ND ATA jpeg> 



<image id= "1" subject= "Liza Hayward' 
position = "(2, 5) (7, 15)" height = "10" width = "5" 
x/image> 

<image id= "2" subject= "J«iBX^Bill"X/image> 
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Fig. 3 

12 



EP1 150 215 A2 




13 



EP1 150 215 A2 




14 



EP1 150 215 A2 



(^^^^ START 



Generate a list of 
metadata labels. 



601 



600 



FIG. 6[a:b] 



617 



619 



Default sized bounding 
box is placed. 



Bounding box is 
resized by the user. 



621 



Yes 




Generate a plurality of 
icons. 



Associate each 
metadata with at least 
one of the icons 



Render an image. 



609 




•^o /Automatic 
subject 



Detect subjects 
automatically and place 
bounding boxes. 



Bounding boxes are 
resized by the user. 



615 



New 

Yes oounding box^ 
to be manually 
placed ? 



No 



603 



605 



607 



611 



613 



15 



EP1 150 215 A2 



Select at least one of the 
icons and drag the icon 
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